Piloting Copilot and Codex: Hot Temperature, Cold Prompts, or Black Magic?

Language models are promising solutions for tackling increasing complex problems. In software engineering, they recently attracted attention in code assistants, with programs automatically written in a given programming language from a programming task description in natural language. They have the potential to save time and effort when writing code. However, these systems are currently poorly understood, preventing them from being used optimally. In this paper, we investigate the various input parameters of two language models, and conduct a study to understand if variations of these input parameters (e.g. programming task description and the surrounding context, creativity of the language model, number of generated solutions) can have a significant impact on the quality of the generated programs. We design specific operators for varying input parameters and apply them over two code assistants (Copilot and Codex) and two benchmarks representing algorithmic problems (HumanEval and LeetCode). Our results showed that varying the input parameters can significantly improve the performance of language models. However, there is a tight dependency when varying the temperature, the prompt and the number of generated solutions, making potentially hard for developers to properly control the parameters to obtain an optimal result. This work opens opportunities to propose (automated) strategies for improving performance.Comment: 14 pages, 3 Figures (not counted the subfigures), 10 Table

Untangling Spaghetti of Evolutions in Software Histories to Identify Code and Test Co-evolutions

International audienceVersion Control Systems are key elements of modern software development. They provide the history of software systems, serialized as lists of commits. Practitioners may rely on this history to understand and study the evolutions of software systems, including the co-evolution amongst strongly coupled development artifacts such as production code and their tests. However, a precise identification of code and test co-evolutions requires practitioners to manually untangle spaghetti of evolutions. In this paper, we propose an automated approach for detecting co-evolutions between code and test, independently of the commit history. The approach creates a sound knowledge base of code and test co-evolutions that practitioners can use for various purposes in their projects. We conducted an empirical study on a curated set of 45 open-source systems having Git histories. Our approach exhibits a precision of 100 % and an underestimated recall of 37.5 % in detecting the code and test co-evolutions. Our approach also spotted different kinds of code and test co-evolutions, including some of those researchers manually identified in previous work

Co-evolution of OCL constraints with evolution of metamodels

Le paradigme ingénierie-dirigée par les modèles (Model-Driven Engine MDE) encourage l'utilisation des modèles et des langages de modélisation durant le processus de développement afin d'améliorer la qualité et la productivité. Les métamodèles sont des composants essentiels de tout écosystème de langage de modélisation afin de définir les aspects structurels d'un domaine d'activité. En complément, l'Object Constraint Language (OCL) est utilisé pour spécifier en détail les aspects du domaine d'activité, par exemple plus de 750 contraintes viennent avec le métamodèle UML. Malheureusement, les métamodèles sont constamment soumis aux changements et évolution qui affectent les contraintes OCL définies qui peuvent avoir besoin d'être co-évolué en conséquence. Bien que plusieurs approches aient été proposées pour détecter les changements de métamodèle lors de l'évolution et de les utiliser par la suite pour co-évoluer les contraintes OCL. Ils ne peuvent toujours pas détecter une trace d'évolution complète et correcte du métamodèle tout en proposant une résolution unique par contrainte OCL impactée alors que des résolutions multiples et alternatives peuvent être appliquées. Dans cette thèse, nous proposons une approche pour détecter les changements de métamodèle lors d'une évolution, tout en visant la complétude et une haute précision. Notre approche de détection considère les changements atomiques et complexes au cours de l'évolution. Par ailleurs, nous proposons une approche dédiée à la co-évolution des contraintes OCL tout en tenant compte des résolutions alternatives et à veiller à ce que seules les résolutions appropriées sont proposées à l'utilisateur pour chaque contrainte OCL impactée. Notre validation montre d'une part que le rappel (recall) de 100% est toujours atteint dans nos cas d'études avec une précision moyenne de 70,75%, qui est encore améliorée par nos heuristiques jusqu'à 91% et 100% dans certains cas. D'autre part, une moyenne de 92% et 93% respectivement syntaxiquement et sémantiquement correcte de co-évolution d'OCL sont atteintes dans nos études de cas. Les deux approches sont implémentées sous forme de plugins pour l'IDE Eclipse, un environnement de développement très répandu pour les développeurs de logiciels. Les plugins sont testés par nos partenaires industriels du projet ANR MoNoGe. Cette thèse a abouti à huit articles publiés et trois autres articles qui sont en cours de soumission / phase de révision.Model-Driven Engineering (MDE) paradigm promotes the use of models and modeling languages during the development process aiming at a better quality and productivity. Metamodels are core components of any modeling language ecosystem to define structural aspects of a business domain. As a complement, the Object Constraint Language (OCL) is used to specify detailed aspects of the business domain, e.g. more than 750 constraints come with the UML metamodel. Unfortunately, metamodels are subject to a constant change and evolution which impacts the defined OCL constraints that may need to be co-evolved as well. Although several approaches have been proposed to detect metamodel changes during evolution and to use them to co-evolve OCL constraints. They still cannot detect a complete and a correct evolution trace of the metamodel while proposing a unique resolution per impacted OCL constraint whereas multiple and alternative ones can be applied. In this thesis, we propose an approach to detect metamodel changes during evolution while aiming at completeness and high precision. Our detection approach considers both atomic and complex changes during evolution. In addition, we propose a dedicated approach to co-evolve OCL constraints while considering alternative resolutions and ensuring that only the appropriate resolutions are proposed to the user for each impacted OCL constraint. Our validation shows on the one hand that a 100% recall is always reached in our case studies with an average precision of 70.75%, which is improved by our heuristics up to 91% and 100% in some cases. On the other hand, an average of 92% and 93% of respectively syntactically and semantically correct OCL co-evolution are reached in our case studies. Both approaches are implemented as plugins for the Eclipse IDE a wide-spread development environment for software developers. The plugins are under test by our industrial partners in the ANR MoNoGe project. This PhD resulted in eight published papers and three other papers are currently under submission/revision phase

Detecting and Exploring Side Effects when Repairing Model Inconsistencies

International audienc

Reporting on a Survey on Approaches to Co-Evolution of Metamodels and Models

International audienc

Transforming Abstract to Concrete Repairs with a Generative Approach of Repair Values

International audienceSoftware models, often comprise of interconnected diagrams, change continuously, and developers often fail in keeping these diagrams consistent. Detecting inconsistencies quickly and efficiently is state of the art. However, repairing them is not trivial, because there are typically multiple model elements that need to be repaired, leading to an exponentially growing space of combinations of repair choices. Despite extensive research on consistency checking, existing approaches either provide abstract repairs only (i.e., identifying the model element but failing to describe the change), which is not satisfactory. This paper presents a novel approach that provides concrete repair choices based on values from the inconsistent models. Thus, our approach first retrieves repair values from the model, turn them to repair choices, and groups them based on their effects. This grouping lets our approach explore the repair space in its entirety, providing quick example-like feedback for all possible repairs. Our approach and its tool implementation have been empirically assessed on 10 case studies from industry, academia, and GitHub to demonstrate its feasibility and scalability. A comparison with three versioned models shows that our approach identifies useful repair values that developers have chosen

Approaches to Co-Evolution of Metamodels and Models: A Survey

Modeling languages, just as all software artifacts, evolve. This poses the risk that legacy models of a company get lost, when they become incompatible with the new language version. To address this risk, a multitude of approaches for metamodel-model co-evolution were proposed in the last 10 years. However, the high number of solutions makes it difficult for practitioners to choose an appropriate approach. In this paper, we present a survey on 31 approaches to support metamodel-model co-evolution. We introduce a taxonomy of solution techniques and classify the existing approaches. To support researchers, we discuss the state of the art, in order to better identify open issues. Furthermore, we use the results to provide a decision support for practitioners, who aim to adopt solutions from research

On the Power of Abstraction: a Model-Driven Co-evolution Approach of Software Code

International audienceModel-driven software engineering fosters abstraction through the use of models and then automation by transforming them into various artefacts, in particular to code, for example: 1) from architectural models to code, 2) from metamodels to API code (with EMF in Eclipse), 3) from entity models to front-end and back-end code in Web stack application (with JHispter), etc. In all these examples, the generated code is usually enriched by developers with additional code implementing advanced functionalities (e.g., checkers, recommenders, etc.) to build a full coherent system. When the system must evolve, so are the models to regenerate the code. As a result, the developers' enriched code may be impacted and thus need to co-evolve accordingly. Many approaches support the co-evolution of various artifacts, but not the co-evolution of code. This paper sheds light on this issue and envisions to fill this gap. We formulate the hypothesis that the code co-evolution can be driven by the model changes by means of change propagation. To investigate this hypothesis, we implemented a prototype for the case of metamodels and their accompanying code in EMF Eclipse. As a preliminary evaluation, we considered the case of the OCL Pivot metamodel evolution and its code co-evolution in two projects from version 3.2.2 to 3.4.4. Preliminary results confirms our hypothesis that model-driven evolution changes can effectively drive the code co-evolution. On 562 impacts in two projects' code by 221 metamodel changes, our approach was able to reach the average of 89% and 92,5% respectively of precision and recall

Supporting A Flexible Grouping Mechanism for Collaborating Engineering Teams

International audienceMost engineering tools do not provide much support for collaborating teams and today's engineering knowledge repositories lack flexibility and are limited. Engineering teams have different needs and their team members have different preferences on how and when to collaborate. These needs may depend on the individual work style, the role an engineer has, and the tasks they have to perform within the collaborating group. However, individual collaboration is insufficient and engineers need to collaborate in groups. This work presents a collaboration framework for collaborating groups capable of providing synchronous and asynchronous mode of collaboration. Additionally, our approach enables engineers to mix these collaboration modes to meet the preferences of individual group members. We evaluate the scalability of this framework using four real life large collaboration projects. These projects were found from GitHub and they were under active development by the time of evaluation. We have tested our approach creating groups of different sizes for each project. The results showed that our approach scales to support every case for the groups created. Additionally, we scouted the literature and discovered studies that support the usefulness of different groups with collaboration styles